Prototyping Virtual Data Technologies in ATLAS Data Challenge 1 Production

A worldwide computing model, embracing a global data and computation infrastructure, is emerging to answer the LHC computing challenges. A significant fraction of the ATLAS Data Challenge 1 (DC1) was performed in a Grid environment. For efficiency of the large production tasks distributed worldwide, it is essential to provide shared production management tools comprised of integratable and interoperable services. To enhance the ATLAS DC1 production toolkit, we introduced and tested a Virtual Data services component in the data management architecture for distributed production in ATLAS DC1. For each major data transformation step identified in the ATLAS data processing pipeline (event generation, detector simulation, background pile-up and digitization, etc) the Virtual Data Cookbook (VDC) catalogue encapsulates the specific data transformation knowledge and the validated parameters settings that must be provided before the data transformation invocation. Because Virtual Data technologies were in the prototyping stage at the start of DC1, the data volume allocated for production tests of the virtual data system was limited to about one fifth of all the DC1 data. To provide for local-remote transparency during DC1 production, the VDC database server delivered in a controlled way both the validated production parameters and the templated production recipes for thousands of the event generation and detector simulation jobs around the world, simplifying the production management solutions. The major benefit of Virtual Data technologies was demonstrated by simplifying the management of the parameter compositions that were different for each of the more than two hundred datasets produced in DC1. Significant reduction in the parameter management overhead enabled successful processing of about half of all the DC1 datasets (representing 20% of the data) using the VDC services. Another benefit of Virtual Data Cookbook technologies is the simplification of the data reprocessing step. We have found it useful to distinguish (both conceptually and in the production system design) the data required before the invocation of the transformation from the information collected after the data transformation completion – data provenance. We further envision that templated recipe catalogues (experiments’ “cookbooks”) encapsulating production gurus’ knowledge in the ‘provender’ data that are necessary before the data transformation can be invoked will be integrated in a coherent system utilizing the Chimera technology from the GriPhyN project. Chimera system eliminates the ‘manual’ tracking of the data dependencies between separate production steps and enables multi-step compound data transformations on-demand.